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Stochastic  Stability  and  the 
Design  of  Feedback  Controls 
H.  Kushner 


1.  IHTRODUCTION 

The  object  of  this  paper  is  to  describe  the  stochastic  extensions  of 
the  various  techniques  for  using  the  second  method  of  Liapunov  to  aid  ••.he 
construction  and  analysis  of  feedback  controls  [1-8].  The  method  appejirs 
to  be  useful  for'  design  and  analysis,  although  it  is  too  early  to  make  a 
final  judgment.  Much  depends  on  future  success  in  finding  suitable  Liapunov 
functions,  and  understanding  the  relationship  between  the  loss  function  and  the 
desired  behavior  of  the  control  system. 

The  deterministic  methods  have  been  motivated  by  considerations  of  the 
following  nature:  Consider  the  optimal  control  problem  with  control  u, 
and  system 


x  =  f(x,  u) 


and  cost 


CU(x)  =  /T k(x,  u)dt,  CU(  86)  =  0 
o 

where  t  is  the  time  of  contact  with  c8,  the  boundary  of  a  target  set 
S.  The  minimum  cost 

C(x)  =  min  CU(x), 

~  u 
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is  achieved  by  u  =  w.  If  C(x)  is  sufficiently  differentiable,  then  the 
Hamilton- Jacobi  equation 

dC(x)/dt  =  C*x(x)f(x,  w)  =  -k(x,  w) 

is  satisfied  (where  *  is  transpose  and  Cx  is  the  gradient  of  C), 
and  w  is  the  u  minimizing 

[C^xJfCx,  u)  +  k(x,  u)]. 

In  lieu  of  attempting  to  solve  this  problem,  an  alternative  procedure 
has  suggested  itself  to  many  authors  (e. g. [l]-[8]) .  Choose  a  Liapunov 
function  V(x),  and  some  u(x)  so  that  the  system  has  suitable  stability 
properties,  and  compute  (X  is  the  state  space) 


V^(x)f(x,  u)  =  -  k^x,  u). 


where  k^(x,  u)  ^  0  in  X  +  &S.  X  is  the  state  space. 

A  comparison  of  k^x,  u)  and  k(x,  u)  can  yield  useful  information;  e.g., 
whether  V(x)  is  greater  or  less  than  C(x)  t  or  stability  properties  of 
the  controlled  system,  the  nature  of  the  problem  for  which  V(x)  and  u 
are  optimum,  and  whether  some  other  calculatable  control  would  minimize  the 
cost  CU(x),  etc. 
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Similar  results  are  achievable  in  the  stochastic  situation  Stochastic 
stability  seems  to  be  a  more  complicated  subject  than  its  deterministic 
counterpart,  since  the  corresponding  Liapunov  functions  do  not  decrease 
monotonically  for  each  sample  function.  The  effect  of  controls  on  the 
statistical  behavior  of  the  system  can  be  made  rather  explicit  in  terms  of 
a  reduction  of  a  bound  on  the  probability  of  arbitrary  deviations  in  the 
sample  paths  before  hitting  dS. 

In  part  II  several  comparison  and  optimality  theorems  are  proved.  In 
part  HE  the  theorems  are  applied  to  the  problem  of  choosing  and  analyzing 
the  effect  of  feedback  controls  for  several  stochastic  systems. 

2.  THE  SYSTEM  TO  BE  CONTROLLED 

The  object  to  be  controlled  is  represented  by  the  vector  stochastic 
differential  (Ito)  equation 

(la)  dx  =  f(x,  u)dt  +  o(x,  u)dz, 

by  which  is  meant  (using  the  Ito  [93  interpretation  of  the  stochastic  integral) 

t  t 

(lb)  xt  =  xq  +  /  f(xt,  u(xt))dt  +  /q  ff(xt,  u(xt))dzt. 

z  is  a  vector  Wiener  process  with  independent,  components,  z  is  commonly 
called  white  Gaussian  noise; 

(lc)  x  =  f(x,  u)  +  c?(x,  u)z  . 

f  is  a  vector  with  components  f^,  and  cr  is  a  matrix  with  components 

a...  The  process  x.  is  confined  to  X, 
ij 
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Without  the  control  parameter  u,  the  meaning  of  (lb)  and  the  conditions 
under  which  a  solution  (a  stochastic  process)  exists  and  is  unique  is  discussed 
in  [9],  [10].  To  be  secure  in  the  mathematical  development  we  assume  these 
conditions.  Let  ||  ||  be  the  Euclidean  norm.  For  some  finite  positive  K,  let 

||f(x  +  a,  u  +  0)  -  f(x,  u)||  g  Kjjajj  +  Kj|p|| 

(2)  lla(x  +  a,  u  +  p)  -  o(x,  u) ||  g  Kjjajj  +  Kj|p|| 

||f(x,  u)||  g  K[1  +  j|x}|2  +  ||uj|2]^ 

(5)  II u(x  +  a)  -  u(x)|j  g  Kjjaj| 

A  control  satisfying  (3)  is  termed  admissible.  (3)  implies  continuity  of  u(x). 
Note  that  n  =  sign  x  is  not  admissible.  Since  the  K  in  (3)  can  be  large, 
admissibility  is  probably  not  a  serious  restriction. 

In  certain  cases,  our  results  are  valid  if  (2)  and  (3)  replaced  by 
local  Lipshitz  conditions.  This  is  the  case  when  the  trajectories  have  appro¬ 
priate  stability  properties  (e.g.,  when  the  origin  is  stable  w.p.l  in  the  sense 
of  [15]). 

The  primary  attractions  of  the  model  (l)  are  that  it  represents  a  rather 
large  class  of  Markov  processes  with  continuous  sample  paths,  there  is  a  large 
body  of  theory  concerning  it,  and  it  seems  that  many  physical  problems  can  be 
modelled  by  it.  The  question  of  modelling  will  not  be  discussed.  The  identifi¬ 
cation  of  particular  forms  of  (l)  with  particular  physical  problems  is  still  an 
open  problem  in  general  (especially  in  the  non-linear  case).  (Some  interesting 
results  in  [11]  clarify  some  of  the  questions  of  modelling.) 

For  each  integer  r,  define  the  stochastic  process 

(4)  Vl  -  xn  +  u(xn))A  +  °<V  "<xn»6V 

where  6zn  =  z((n+l)A)  -  z(nA),  and  define  xr(t)  =  x£  in  the  interval 

>  t  ^  nA.  Then,  for  a  suitable  sequence  of  A  ~ *0 ,  we  have  x^*(t)— >x(t) 
with  probability  one  for  each  t,  where  x(t)  is  the  solution  to  (1). 
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Some  facts,  to  be  used  later,  will  be  quoted.  If  u(x)  is  admissible, 
x  is  continuous  with  probability  one,  and  is  a  Markov  process;  i.e.,  for 
any  measurable  set  A  in  X, 


(5)  Pfxt+s  «  A  I  V  0  -  *1  -  p[xt+s  6  A  1  *t]» 

where  the  bar  |  denotes  conditional  probability.  A  major,  relatively 

recent,  development  in  probability  theory  is  the  analysis  and  extensive 

use  of  the  concept  of  random  time  (see  [10],  [12],  [13]  for  details).  An 

example  of  a  random  time  is  the  first  time  that  x^  leaves  an  open  set  A; 

t  =  min  [t:  x  ^  A) .  t  is  a  random  variable.  Loosely  speaking,  whether 

or  not  the  event  [t  <  t}  has  occured  (in  the  example,  whether  x  has 

s 

left  A  by  time  t)  can  be  determined  by  observations  on  the  x  process 

s 

up  to  and  including  time  t.  (The  set  [t  <  t]  is  in  the  o-field 
dt';e:  r!ned  by  x  ,  s  5  t.) 

The  significance  to  control  applications,  of  the  concept  of  random 
time,  will  be  seen  in  the  sequel.  If  the  process  x^.  is  confined  to  a 
set  X  which  is  compact,  and  if  u  is  admissible,  the  process  x^.  is,  in 
fact,  a  strong  Markov  process.  A  strong  Markov  process  has  the  Markovian 
property  relative  to  random  times.  Let  t  be  a  random  time,  then 


P[x  €  A 

L  T+S 


8* 


6  §  t]  =  P[x 


T+S 


e  A 


g- 


For  example,  let  x^.  start  in  an  open  set  B,  let  t  be  the  least  time 
of  leaving  B,  then  for  any  non  random  s  the  probability  that  x  e  A 
given  x^  and  the  paths  up  to  t  equals  the  probability  given  only  x^.. 
The  strong  Markov  property  is  proved  in  [10]. 
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3.  THE  CONTROL  PROBLEM 

The  process  x  is  defined  in  a  set  X  in  a  Euclidean  space.  There 
is  a  set  S  in  X  given,  and  the  main  object  of  the  control  is  to  trans¬ 
fer  xq  =  x  to  8  S  in  finite  average  time.  In  certain  cases,  infinite 
average  times  vill  be  allowed.  The  proofs  of  the  theorems  we  require  as¬ 
sume  that  X  is  compact  (e.g. ,  the  proof  of  (8)  for  the  operator  LU). 

This  does  not  seem  to  be  a  restriction  from  the  practical  point  of  view 
since  X  may  be  as  large  as  desired.  We  may  stop  the  process  upon  leaving 
some  very  large  set,  and  estimate  the  probability  of  this  event  by  (10). 

Also,  to  each  u  and  initial  point  xq  =  x,  there  is  the  associated  cost 

(6)  CU(x)  =  E*  /^U  k(xt,  ut)dt 

E^  is  the  expectation  and  is  the  random  time  of  arrival  at  d  S, 

(provided  that  il  is  defined)  and  k(x,  u)  is  continuous  and  non-negative, 
and  is  referred  to  as  the  loss. 

Define  C(x)  =  min  CU(x),  provided  that  the  minimizing  u  is  admissible. 
Part  of  the  control  problem  is  the  comparison  of  Cu(x),  and  E^  for 
various  controls.  Various  restrictions  may  be  placed  on  the  control;  it 
may  be  bounded,  or  its  functional  form  may  be  restricted;  e.g. :  it  may  be 
allowed  to  be  a  function  of  some,  but  not  all,  components  of  x.  Some  sta¬ 
bility  properties  may  also  be  of  interest;  e.g.,  an  estimate  of  the  probability 
that  x^  ever  leaves  some  set  X',  if  xq  is  in  X’,  or  some  other  quali¬ 
tative  information  cn  the  random  paths. 

A  number  of  relevant  results  and  examples  on  stability  are  in  [l4]-[l6], 
[17]  is  concerned  with  ergodic  properties  of  the  processes  and  utilizes  cer¬ 
tain  properties  of  stochastic  Liapunov  functions. 


4.  OTHER  MATHEMATICAL  PRELIMINARIES 


Let  u  be  admissible.  Define  the  operator 


(7) 


S.  .(x,  u) 


a2 

cbcTSxT 


S.  .=  E  cr.,  a., 
ij  k  ik  jk 

LU  is  the  differential  generator  of  the  x^  process,  with  control  u. 

We  say  that  V(x)  is  in  the  domain  of  Lu  (V(x)  e  D(L))  if  V(x)  is  a 
non-negative,  scalar  valued  function  with  continuous  second  derivatives 
and  the  sets  {x  :  V(x)  ^  c}  are  compact  and  connected,*  for  all  c  less 
than  some  cq  >  0.  Such  a  V(x)  will  also  be  called  a  Liapunov  function, 
or  a  Liapunov  function  in  a  region  R,  if  l'V  £  0  in  R  for  the  given 
u.  Note  that  LuV(x)  =  dv(x)/dt  =  V'(x)f(x,  u)  in  the  deterministic 
case. 

Since  X  is  compact  and  x^  is  a  strong  Markov  process, 

Dynkin's  formula  [10] 


(8)  E*  V(xt)  -  V(x)  =  E*  fXQ  L^x^ds 

holds  for  all  random  times  t  with  E^  t  <  ».  (8)  underlies  many  of  the 

results  of  the  sequel.  It  says,  in  effect,  that  V(x)  is  the  average 
value  of  the  integral  of  the  ’stochastic  derivative'  L^VCx).  The  com¬ 
pactness  of  X  and  the  finiteness  of  E^t  are  important  in  establishing 
its  validity.  In  other  cases  the  operator  LU  f  for  which  (8)  is  valid, 
will  be  an  extension  of  (7),  but  this  is  beyond  our  purpose. 

Let  V(x)  be  in  D(LU)  in  the  region  R  €  X, 


*The  domain  of  Ln  is  obviously  larger  than  our  D(L),  but  D(L)  suffers 
less. 
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R  =  {x  :  V(x)  <  X}  -  {x  :  V(x)  ^  Xq),  X  >  Xq 

Br  =  (x  :  v(x)  =  x,  v(x)  =  xo). 

Let  L^V  <  0  in  R  and  L^x)  £  0  on  8  R.  Let  be  the  random 
time  to  8  R,  starting  at  x  e  R.  It  can  be  proved,  using  the  continuity 
of  x^  and  L^x),  and  the  compactness  of  X,  that  tu  will  exist 
(although  E^u  may  not  be  finite),  and  that  the  integral  in  (8)  is  defined 
and  finite,  and  that  (8)  is  valid.  (See  [15]).  Since  x^  is  continuous 
with  probability  one,  x^.  cannot  leave  R  without  toucJ  Ing  d  R  (with 
probability  one).  We  have 


V(x  )  =  \jP[  sup  V(x  )  £  X  ] 

x  T  t  lt§0 

u 

+  X  P[  sup  V(x  )  §  X]  . 

t  *t£0 
u 

0,  and  noting  that  the  integral  in  (8)  is  non-positive  in  R  +  8r, 

P[  sup  V(x  )  *  X]  =  (V(x)  -  E V  U  L^xJdtJ/X 
T  ItlO  ° 

*  V(x)/X. 

(10)  can  also  be  derived  by  showing  that  V(y^)  is  a  non-negative 
super  martingale,  where  y^  is  the  x^  process  stopped  at  time  tuj  then 
the  inequality  (10)  is  the  non-negative  super  martingale  inequality.  Both 
methods  may  be  used  when  time  is  discrete.  If  LUVn(x)  ^  0  in  R  for  any 
real  number  n  £  1,  then 


(9) 

Letting  Xq  = 
(lO't 
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(11)  P  [  sup  v(x  )  n]  ^  V*(x)/Xn 

T  *t*o 

u 

which  is  an  improvement  over  (10), 

In  general,  we  will  try  to  improve  (10)  by  finding  the  maximum  n 
for  which  LUVn(x)  £  0  in  R.  This  method  is  rot  generally  the  best  for 
obtaining  probability  bounds  on  the  behavior  of  components  of  x. 


Part  II.  COMPARISON  AMD  OPTIMALITY  THEOREMS 

It  is  always  assumed  that  X  is  compact,  k(x,  u)  £  0  and  continuous, 
and  that  (2),  (3)  are  satisfied.  The  purpose  of  the  theorems  is  to  allow 
a  comparison  of  the  costs  and  stability  properties  resulting  from  the  use 
of  different  controls,  and  to  obtain  upper  lower  bounds  on  C(x)  with¬ 
out  actually  solving  the  minimization  problem.  The  symbols  tq,  are 

the  random  times  to  transfer  xq  =  x  (in  some  given  initial  set)  to  ds, 
uhe  boundary  of  the  target  set  S,  in  the  cases  of  no  control,  and  control 
u,  respectively. 

The  theorems  use  the  assumption  E^t  <  ».  When  LUV(x)  <0  in  X  -  S 
and  LUV(x)  £  0  on  dS  and  does  not  depend  on  time,  the  finiteness  assumption 
may  be  dropped.  The  modification  will  be  used  occasionally  in  the  examples. 

It  is  usually  of  little  consequence,  since  a  slight  enlargement  of  the  target 
set  will  usually  assure  that  E^Tu  <  ** 


-lo¬ 


in  X  -  S.  Then 

(15)  Vx(x)  >  C(x) . 

Also,  for  any  X  >  0, 

C1^)  PC  sup  V  (x  )  *  X]  *  EVn (x)/X. 

t  1  Z  1 

u 

If  there  is  a  Vg(x)  in  D(L)  with  V2(dS)  =  0  and,  for  all  admissible  u, 
U7)  L^^x)  +  k(x,  u)  >  0, 

in  X  -  S,  'chen 
(IS)  V2(x)<C(x). 

(In  the  even"  that  there  is  a  non- admissible  control  for  which  the  problem 

has  a  meaning,  and  which  minimizes  CU(x),  then  the  first  part  of  the  theorem 
still  holds.) 

Proof; 

(8)  may  be  applied  to  Vn(x)  and  t  .  Thus, 

1  u  ’ 

VjM  -  t  >  >  eKu  k(x>  u>dt  -  £«• 

U 

Since  E*  ru  <  »,  x^  is  on  c)S  w.p.l.  Since  X  is  bounded  and 

u 

V1(SS)  =  0,  we  have  E*  V^x  )  =  0,  and  (15)  follows. 

Tu 

Since  w  is  admissible,  and  Vg(x)  is  in  D(L)  and  Ewt  <  the  ap- 

X  w 

plication  of  (8)  to  Vg(x)  and  t yields 

V2(x)  -  V,(x  )  <  E*  /  *  k(x,  w)ds  ,  C(x), 
w  ~~ 

w 

Ev  Vg(xT  )  =  0  by  a  repetition  of  a  former  argument.  (l6)  follows  from  (10). 


rr. 


r' 


~~r  -mb**  - 


Corollary  1 


} 

I 

1 
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Let  the  optimal  admissible  control  exist,  and  let  V(x)  satisfy  the  condi- 

W  11 

tions  on  V,  (x).  Let  u  and  w  be  admissible  controls,  E  t  <  »,  E  t  <  », 
-  1'  •  -  -  - - >  x  w  1  x  u  7 

and,  for  all  such  u, 

L“v(x)  +  k(x>  u)  *  0 
vith  equality  when  u  =  v.  Then 
V(x)  =  C(x) 
and  w  is  optimal. 

Proof: 

The  statement  follows  from  Theorem  1,  by  setting  V(x)  =  V-^x)  =  V2(x)  and 
replacing  all  >  by 

Remark: 

If  there  is  an  admissible  control  which  is  optimal  and  a  V(x)  satisfying 
the  conditions  of  the  corollary  is  available,  then  the  corollary  partially  jus¬ 
tifies  the  usual  result  of  dynamic  programming;  i.e.,  that  the  optimum  control 
minimizes  (19)  and  that  the  solution  of  (19)  is  V(x)  =  C(x). 

(19)  minfL^x)  +  k(x,  u)]  =  0 

u 

Corollary  2 

(10)  is  valid  and  Is  defined  with  probabi3-ity  one,  when  k(x,  u)  >  0 
in  X  -  S.  Under  this  condition,  the  condition  on  definiteness  of  the  average 
arrival  times  can  be  dropped,  and  we  have  a  true  optimality  theorem,  (the  stochas¬ 
tic  counterpart  cf  the  Hamilton-Jacobi  equation  theorem  in  [20]), 

Theorem  2 

Let  «,  II  ,T 

Cu(x)  »  e“/*»  [*(x)  +  i(x,  u)]dt 


?**■ 
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where  k£  0  ,  l  ^0  and  l(x,  0)  =  0.  Let  L°  correspond  to  u  =  0.  Let 
E°to  <  °°>  V(x)  D(L)  and  V(  3  S)  =0  and 

L°V(x)  +  k(x)  =  0. 

For  some  u,  let  EUt  <  »  and 

(20)  L^x)  +  k(x)  +  i(x,  u)  <  0. 

Then 

(21)  C°(x)  =  E°  /  °  k(x)dt  >  E *  f  u  [k(x)  +  l(x,  u)]dt  =  Cu(x). 

X  Q  A  O 

If  S  replaces  >  in  (20),  it  does  so  in  (21). 

Proof: 

The  proof  is  essentially  that  of  Theorem  1.  From  (8) 

Ey  /  U  [L^U)  +  k(x)  +  l(x,  u)]dt  <  0  =  E°  /  °|L°V(x)  +  k(x)]dt 

X  q  XU 

-  V(x)  +  eJJv(xt  )  +  CU(x)  <  -  V(x)  +  E°V(xt  )  +  C°(x) . 
u  o 

Since  x  and  x  axe  on  d  S  v.p.l,  and  V(x)  is  bounded  in  X  -  S, 
u  o 

the  theorem  follows. 

Remark: 

Consider  the  special  case 


dx  =  f(x,  u)dt  +  o(x)dz. 
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where 


(22) 


With 


a  does  not  depend  on  u,  and  where  (with 
k(x)  =  -  L°V(x)  =  -  V^(x)f(x,  0)  - 

uf  0, 


V(  d  S)  =  0) 


1  2  ^x) 
5  iyi  ^i^J 


(x). 


(25)  -L"v(x)  =  -Vi(x)f(x,  »)  -  \  ^  «„(*). 

By  Theorem  2,  for  any  u  such  that 

l'V(x)  -  L°V(x)  +  i(x,  u)  SO 
or 

(24)  V^(x)  [f(x,  u)  -  f(x,  O)]  +  i(x,  u)  <  0, 


we  have 


CU(x)  <  C°(x). 

✓ 

Although  the  theorem  states  that  a  control  will  decrease  the  cost 
under  certain  conditions,  accurate  estimates  of  the  decrease  are  usually 
difficult  to  obtain.  Estimates  of  the  effect  of  the  control  on  the  pro¬ 
bability  (l6)  are  readily  available  (see  the  examples).  We  obtain  the 
best  improvement  of  the  value  of  (i 6)  with  the  u  which  minimizes  (20). 
Otherwise,  the  problem  of  selecting  one,  from  among  the  many  controls  which 
may  satisfy  (24),  is  open. 

Theorem  5  gives  a  condition  under  which  t  <  *•  is  assured. 


Theorem  3 
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Let  V(x)  be  in  D(L).  If,  for  some  e  >  0, 

L^x)  =  -k-^x,  u)  ^  -  e 

in  X  -  S,  then  exists  w.p.l  and 

Eu  t  g  V(x)/e  <  oo. 
x  u  N  " 

Let  there  exist  an  optimum  w  and  C(x)  with  loss  function 
k(x,  u).  Let  k(x,  u)  ^  k^(x,  u)  and 

inf  k(x,  u)  ^  e  >  0. 
x,u 


Then 


Ex  \  ~  y(x)/€  <  "• 


Proof. 

Let  t  be  any  random  time  with  E^t  <  oo.  The  first  statement  follows 

from 


V(x)  -  E'VCx  )  =  E u  /T  k  (x,  u)dt  ^  e  EUt 
\  /  x  '  t'  x  o  1'  *  '  x 

and  from  the  boundedness  of  V(x)  in  X.  (If  EUt  =  oo  we  could  increase 

v  '  '  x  u  ’ 

t  until  e  E^t  >  V(x)  in  X.) 


Now,  by  Theorem  1, 
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T 

V(x)  *  C(x)  =  E^  /oW  k(x,  u)  *  e  E\w 

and  the  second  statement  follows.  The  existence  of  a  is  part 

of  the  statement  on  the  existence  of  an  optimum  w. 

Theorem  4  gives  a  method  of  selecting  S  so  that  the  corresponding 
problem  can  be  studied  by  means  of  Liapunov  functions. 


Theorem  4 

Let  u  be  admissible  and  let  V(x)  be  in  D(L).  Define  the  sets 

Ru  =  {x  :  L^x)  ^  0}  and  =  {x  :  V(x)  £  r). 

Let  the  sets  be  non-empty  and  let  RU  be  a  proper  subset  of  S^. 

Let  x  =  x  be  in  X  -  S  and  define  t  as  the  random  time  of 
—  o  -  r  -  u  - 

arrival  at  dS^..  Then 

(25)  EUt  <  «. 

(26)  P[  sup  V(x  )  -  r  £  x]  s  E[v(x)  -  r]/4 

T 

u 

If  w  minimizes  LUV(x)  in  X  -  and  LwV(x)  =  -k(x)  £  -  €  <  0  in 

X  -  then  w  is  the  optimal  control  for  the  loss  k(x)  and  target 

set  Sr.  The  cost  is 

(27)  C(x)  -  V(x)  -  r  =  Ejj  fl*  k(x)dt. 

Also,  if  LUV(x)  +  k(x)  SO  in  X  -  then 
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(26)  V(x)  -  r  ^  Cu(x)  =  /\(x)dt. 

Proof: 

Since  LuV(x)  is  continuous,  and  Ru  is  a  proper  subset  of  S^.  , 
L^x)  £  -  e  <  0,  for  some  €,  in  X  -  S  .  Consequently  (25)  follows 
from  Theorem  5. 

Since-  V(x)  -  0  in  X  -  and  LU[V(x)  -  r]  S  0  in  X  -  S  , 

(26)  follows  from  (10).  The  fact  that  the  u  which  minimizes  LuV(x) 

is  an  optimal  control  for  loss  k(x)  =  -  min  LUV(x)  and  target  d  S  fol- 

u  * 

lows  from  Corollary  1.  (27)  and  (28)  follow  from  Theorem  1  and  Corollary 

1,  by  using  V(x)  -  r  in  their  proofs. 

Discussion: 

For  a  given  Liapunov  function  V(x),  the  control  problem  may  be 
studied  in  several  ways. 

Let  the  loss  be  k(x,  u).  Now  compute  Ru  =  (x  :  l'V  *  0).  Now  choose 
a  x  such  that  S^O  R^  and  check  that  X  -  S^.  is  not  empty.  Check 
that  LUV(x)  +  k(x,  u)  ^  0  in  X  -  S^.  Then,  Theorem  4  says  that,  start¬ 
ing  from  a  point  in  X  -  S  ,  the  total  cost,  CJ(x)  of  transferring 
xq  =  x  to  a  point  on  d  S  is  no  greater  than  V(x)  -  r.  If  I'M*)  + 

+  k(x,  u)  =  0,  then  the  cost  is  V(x)  -  y=  Cu(x). 

u  u-i 

Now  let  u^  and  u  be  given  and  check  that  S^^(R  UR  )  and  that 
X  -  S  is  not  empty.  If  LuV(x)  +  k(x,  u)  £  L  ^V(x)  +  k(x,  u^)  =  0,  then 
the  theorem  says  that  the  cost  of  transferring  x  =  xQ  in  X  -  to  8 
is  no  greater  with  u  than  with  u^.  Theorem  2  may  be  used  to  try  to  find 
improved  controls,  provided  that  V(x.)  and  k(x)  are  given. 
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If,  for  some  S,  two  Liapunov  functions  V^(x)  and  VgCx)  are 
given  with  the  properties  V^(  d  S)  -  ,  and  LUV^(x)  +  k(x,  u)  5  0, 

LlVg(x)  +  k(x,  u)  £  0,  LUV^(x)  <0  in  X  -  S,  then  the  cost  of  trans¬ 
ferring  xq  =  x  to  3  S  is  bounded  by 

v2(x)  -  r2  *  Cu(x)  *  V^x)  -  t-l 

Obviously,  the  cost  of  transferring  to  a  point  interior  to  S  is  no 
less  than  the  cost  of  transferring  to  S  (by  the  continuity  of  x^).  The 
cost  of  transferring  to  a  set  enclosing  S  is  no  greater  than  the  cost  of 
transferring  to  S.  The  observation  yields  bounds  for  terminal  sets  other 
than  the  S. 

Other  forms  of  boundary  conditions  and  loss  functions  and  possible  (for 
example,  in  case  of  instability  we  may  minimize  the  probability  of  being  lost)  and 
will  be  considered  in  the  examples.  Choosing  suitable  V(x)  is,  of  course, 
no  easier  in  the  stochastic  case  than  in  the  deterministic  case.  We  have 
the  double  problem  of  finding  V(x)  so  that  both  k(x,  u)  and  S  are 
suitable. 

In  the  (homogeneous)  deterministic  case  when  V(x)  §  0  with  equality 
implying  x  =  0,  it  is  possible  to  transfer  xq  =  x  to  the  origin.  This 
is  possible  in  the  stochastic  case  (with  probability  one)  if  L^x)  $  0 
with  equality  only  when  x  =  0. 

The  following  theorem  is  useful  for  obtaining  probability  bounds  on  the 
rate  of  convergence  of  x^  to  d  S.  The  quantity  a  may  depend  on  the 
control. 

Theorem  5 


Let  V^x)  be  in  D(L)  and 
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lV(x)  s  -  a  vn(x),  a  >  o, 

in  X  .  S,  Let  t(r)  »  min  [t,  *i],  for  t  non- random.  Then 
P[  »up  V(xJ  *  X]  S  e“a*Vn(x)/Xn, 

T  Itl fl( T  )  Z 

where  x  ■  x  . 

-  o 

Proof 

Modify  the  system  in  S  only,  so  that  L^Cx)  h  -  cN(x)  in  X.  Let 
x1  be  the  modified  trajectory  and  t1  the  time  to  the  origin  for  the 
modified  trajectory.  By  the  continuity  or  ths  paths 

P[  sup  Vn(x')  %  Xn]  t  J’[  sup  v"(x.)  4  Xn]. 

t'  4t4a(  t'  )  >c  fct£«(  T  )  v 

v  '  u  '  u' 

By  Theorem  5  of  [15],  the  left  side  la  leer;  than  e’BQVn(x)/Xn;  if  xQ  ■  x, 
and '-the  proof  is  concluded.  •  - 


UL  EXAMPLES 

Example  1.  Let'*' 


dx.  ■  x0dt 

(29)  1  2 

dxg  ■  (-x  -  x2  +  u)dt  +  o(x)dz 


with 


The  spaces  X  of  the  examples  are  not  compact.  However,  by  letting 
a2  a  0  for  large  ||xj|,  and  confining  x0  *  x  to  some  large,  but  compact, 
set,  the  space  may  be  ccmpactified  with  little  loss  in  generality. 
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a2(x)  -  x2  a2,  y2  <  2 
2  2  2 

k(x,  u)  «  +  Xg  +  u  . 

2 

If  a  <  2  and  u  ■  0,  then  x^  -*  0  with  probability  one  [15].  Owing  to 
this,  the  target  set  may  be  the  origin.  Theorem  2  will  be  applied  to  the  com 
putation  of  a  control.  With  u  ■  0,  there  is  a  positive  definite  quadratic 

form  [15] 

V(x)  .  bux|  +  2^*2^  +  b224 

»u  -  5  +  (2  +  o2)/(2  -  o2) 

”12  *  S  +  °2/<2  *  °2> 

b22  .  2/(2  .  bS) 


L°7(x)  »  -  k(x,  0)  -  -  x2  -  x2 
C°(x)  -  E°  /0°(x2  +  x^)dt  «  V(x) . 


By  Theorem  2  (Eq.  (2U)),  for  any  u  such  that 


(30)  u(dV/dx2)  +  u2  <  0, 
we  have 

(31)  Cu(x)  -  C°(x)  <  0. 
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In  particular, 


(32)  u  =  -  (dv/dx2)/2  =  -  ('b11x1  +  b^x^) 

satisfies  (30).  Although  the  improvement  (31)  is  difficult  to  estimate, 
an  estimate  of  the  stability  improvement  may  be  obtained  with  the  use  of 
(11).  Now,  with  (32),  and  any  real  number  n  £  1, 


lV  ■  nVl’‘1  t|r  +  i:  (-v*2  - 


(33) 


2  2 

+  -5—  [nV  — 7  +  n(n-l)V  (— )  ] 

ax2  ax2 

■  nVn'1[-x2  -4  -  2(bl2x1  +  b22x2)2  +  2(n-l)tr2(b12x1  +  bjjXg)2 


2 

The  middle  entry,  _2(b12Xl  +  b22X2^  ’  is  due  to  the  confcro1*  The  noise 

2  2  1 
contributes  to  all  other  terms.  Also  (x^/V)  s  (b^  -  * 

Since  the  control  contribution  ls^propdrtiohai  to  the  -term  containing"  -*■< 

2 

ct  ,  and  is  of  opposite  sign,  some  cancellation  occurs;  this  cancellation 
increases  the  maximum  value  of  n  for  which  l'V1  is  non  positive  in  X. 
As  n  increases,  the  estimate 


(3*0  P[  sup  V(x.)  £  X]  £  vH(x)/Xn 

t  at£0  £ 
u 

improves.  With  properly  chosen  V(x),  estimates  of  the  form  of  (3*0  can 
yield  useful  information  on  the  effect  of  the  particular  controls.  By 
Theorem  3,  if  S  is  a  set  containing  the  origin  as  an  interior  point,  then 
the  average  time  to  S  is  finite.  In  any  case,  x .  -*  0  w.p.l. 


(x*/v)]. 


A  useful  general  form  is 
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LV  =  nV11-  X[  L°V  +  (LU  -  L°)V  +  (n-1)  S 


(dv/cfc^KdV/dx^ 


S.  .  =  L  u.,  a.. 
lj  k  ik  Jk 


In  our  case, 


(LU  -  L°)V  =  u(dv/dx2)  =  2(b12xi  +  b21X2'U 


Let  numbers  6  >  0  and  e  <  X  be  given,  and  assume  that  x.  =  x  is 
in  {x:  V(x)  ^  e}.  We  will  compute  the 'smallest'  control  which  guarantees 
(according  to  our  estimates  and  method)  that  x^.  -*  0  w.p.l  and 


P[  sup  V(„.)  *>3*6 


t  at5£0 
u 


First  compute  the  least  n  *  1  such  that 


sup  A x)/Xn  =  (e/X)n  =  6. 


'•hen  lV1  *  0  in  X  if 


-x2  -  x2  +  2u(b12x1  +  b22x2>  +  +  b22X2^xl/V)  "  0 


2,  2 


A  suitable  control  can  be  determined  from 


2  2 

2u(b.|pX^  +  =  min[0,  x±  +  x^ 


2(n-l)cr2(bl2x1  +  b22x2)2(x2/V)  J, 


which  always  yields  a  bounded  control  (in  any  compact  set)  (if  +  b22x2  = 

then  u  -  0) . 
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2  2 

Example  2.  Same  as  Example  lt  but  let  cr  (x)  =  a  ,  a  constant.  We 
would  prefer  a  V(x)  such  that  L°V(x)  =  -k(x,  0).  Not  being  able  ^o  find 
such  a  V(x)?  we  select  one  which  yields  an  approximation.  If 

V(x)  =  3/2  x^  +  x^Xp  +  Xg  , 


then 


L°V(x) 


2  2  2 
-x^  -  Xg  +  cr  . 


To  satisfy  the  conditions  of  Theorem  4,  let 
so  {xs  x^  +  Xg  S  <x%  R°  . 


L°V(x)  <0  in  the  complement  of  R°.  Although  RU  con  be  made  smaller  than 
R°,  the  minimum  eigenvalue  will  be  the  same,  and  the  allowable  reduction  in 
the  size  of  the  target  set  may  not  be  appreciable.  Following  the  procedure 
of  Theorem  4, 


06) 


,o 


C°(x)  «  E~  / 
x  '  x  o 


op  P  2 
(x£  •••  -  cr  )dt  =  V(x)  -  r, 


where  tq  is  the  random  time  to  the  assumed  target  set 
S  =  {x«.  V(x)  g  A)  35  R°. 


CU(x)  <  C°(x)  if 
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uc  +  (dV/dx2)u  <  0, 


which  is  satisfied  (and  is  minimum)  if 


(37)  u  =  -(xx/2  +  x2). 


Also 


LUVn(x)  =  nVn-1(x)[-x^  -  x2  +  a2  -  2(x1/2  +  x2)2 

+  2a2(n-l)(x1/2  +  x2)2/v(x)], 

2 

The  -2(x1/2  +  x2)t"  term  is  contributed  by  the  control  (37).  As  in  example 
1.  the  control  improves  stability — in  the  sense  that  the  probability  of  an 
arbitrary  increase  in  V(x^)  (before  absorption  on  dS)  is  decreased. 

The  method  may  be  used  to  obtain  bounds  on  moments. 

2 

Replace  in  (36)  by  a  non-random  variable  t,  let  o  =  0  for  very 

•  2 
large  |j xjj  (so  that  X  is  compact),  and  assume  that  each  F.  xt  converges 

2  2 

to  a  constant  as  t  -*  ®.  Since  x^  +  Xg  is  bounded  in  X,  the  order  of 
integration  may  be  changed  for  any  finite  t.  Then,  (36)  and  the  boundedness 
of  V(x)  in  X  yield  that 

E<xx +  *2> s  11 

2  p 

and  the  limit  converges  to  cr  as  the  point  |jxjt  of  truncation  of  a  goes  to 
infinity. 

Example  3.  Assume  the  system  ,of  example  2,  We  consider  another  type  of 
criteria  by  which  V(x)  may  be  chosen.  Let;  xQ  =  x  =  (0,  x^q).  Then 


-TT’ 
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V(xq)  =  b O2x2o *  yind  a  u  which  wil1  transfer  x2  to  some  small  value 

02  p 

p  and  such  that,  for  a  given  8  and  e  >  Xg^, 

P[  sup  x?  *  e]  $  8. 

t  £t?sO  Z 
u 

Let  LUVn(x)  <  0  for  x2  >  £2,  and  let  he  well-defined. 

Any  quadratic  form  in  two  variables  may  be  written  as 

(38)  V(x)  =  b'  Xg  +  (b^  +  b12x2)2/lDu 

b’  =  (b22  -  b12/b11), 

where  the  first  term  of  V(x)  is  positive  definite,  and  the  second  is 
positive  semi-definite.  Since 

(59)  pf  SUP  b’x2  *  X1  g  p[  SUP  V(x)  *  4]  *  (b_pXp  )n/>,n, 

t  iteo  t  st^o  22  20 

u  u 

where  ?./b*  =  €,  it  seems  reasonable  to  use  the  positive  definite  quad¬ 
ratic  form  with  the  maximum  value  of 

b  /  bgg  =  1  -  bi2/bllb22^ 

provided,  of  course,  that 

(40)  I.V(x)  <  0  for  x|  >  $2 

and  a  suitable  n.  The  problem  suggests  that  we  seek  a  V(x)  such  that 
o  2 

L  V(x)  =  -Xg  +  constant.  Thus,  let 
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V(x)  =  (x2  +  x2)/2, 

L“v(x)  =  -x2  +  a2/2  +  uxg . 

2  2 

If  p  <  cr  /2,  then  the  use  of 

(^)  u  =  -x2(cr2/ 2  -  £2)/£2,  p2  g  x2  s  cr2/ 2 

=  0  otherwise 

assures  that  (^0)  is  satisfied  for  n  =  1.  Thus,  there  is  a  control  for 
2  2. 

which  Xg  =  {3  is  attainable.  Note  also  that  b*/b22  is  maximum.  If 
k* /^22  were  no^  maximum,  then  either  some  systematic  procedure  for  maximiza¬ 
tion  would  be  followed,  or  else  several  V(x)  would  be  tried  and  compared. 

To  complete  the  analysis,  find  the  least  nil  for  which 

(b22X20 /eb‘ ^  =  6 

and  choose  the  most  convenient  u  for  which 

2  2 

-iL-n,  »  __n—  1-  2  2  (n-l)cr  x2 

L  v  (x)  =  nv  [-x2  +  a  +  — g - 5-  +  uxp] 

(x1  +  x2) 

is  negative  in  the  desired  region  x^  £  p2. 

There  are,  of  course,  similar  procedures  for  more  general  initial  condi¬ 
tions.  The  quadratic  forms  may  be  chosen  by  selecting  the  non-constant,  non¬ 
positive  quadratic  part  of  L°V(x),  and  solving  for  V(x). 

Other  forms  of  experimentation  with  the  type  of  quadratic  form  is  pos¬ 
sible^  e.g. ,  choose  a  control  first  (say,  of  an  arbitrary  linear  form  with 
coefficient  to  be  determined),  then  choose  x'Bx,  so  that  the  target 
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{ x:  x'Bx  e  X)  is  of  some  useful  shape,  and,  finally,  compute  the  control 
coefficients. 

Remark: 

Generally,  the  Liapunov  functions  V^x)  do  not  give  the  best  probabi¬ 
lity  bounds  on,  say,  the  excursions  of  some  component  |  xj ,  since  it 
couples  the  effects  of  the  various  components  of  x  more  than  is  necessary. 
For  example,  instead  of  choosing  n  =  2,  a  suitably  chosen  homogeneous 
positive- definite  quartic  form  will  usually  yield  better  estimates  on 
the  probabilistic  behavior  is  being  investigated.  The  powers  of  the  quadratic 
form  are  used  here  purely  for  numerical  simplicity. 


Example  4.  Let 

dx  =  (Ax  +  Cu)dt  +  adz 
k(x,u)  =  F(x)  +  g(u) 

F(x)  =  Z  F2  (x), 

where  F2^(x)  is  a  homogeneous  positive  definite  form  of  order  2i,  and  A 
is  stable.  By  a  theorem  of  Liapunov  [18],  if  a  =  0,  and  u  =  0,  there  is 
a  homogeneous  positive  definite  function  Vg^(x)  of  2i-tb  order,  with 
V2^(x)  =  -F2i(x).  When  a  is  a  constant  matrix  not  identically  zero. 


I«0V2i(x)  =  -F2i(x)  +  i_i)  (x) 


^(i-l)^ 


1 

2 


z 


^21 

3x7  cbc 
J  m 


S 

3® 
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Qq  =  constant. 

S .  =  Z  a..CT  . 

jm  i  ji  mi 


i  ^  °t  is  a  homogeneous  non-negative  definite  form  of  order  2i. 
A  Liapunov  function 


(42) 


v(x)  =  |  v2i 


with 


(43)  L°V(x)  =  -F(x)  +  Qq 

is  easily  determined:  set  ct  =  0  and  solve,  by  Liapunov' s  theorem, 
^2n^  =  "F2n^* 

and,  in  general,  for  the  case  0  <  i  <  n, 

^21  =  ~F2i^  ~ 

If  the  target  set  S  =  {x:  V(x)  =  r)  includes  (x:  F(x)  2  Qq},  then 
C°(x)  =  E°  /o°( F(x)  -  Q0)dt  =  V(x)  -  r. 

By  Theorem  2,  if 


LUV(x)  +  g(u)  <  L°V(x) , 
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then 


CU(x)  =  V?  /0U(F(x)  +  g(u)  -  Q0)dt  <  C°(x) . 

For  the  deterministic  problem,  this  approach  was  investigated  in  con¬ 
siderably  more  detail  in  [8]. 

Example  5.  Let 

(44)  dx  =  (Ax  +  u)dt  +  adz 

k(x,  u)  —  -  p 

2„ 

A'  +  A  =  0,  a.  .  =  a  o. 

2 

and  u*  u  =  p  .  The  target  set  is  to  be  a  sphere  about  the  origin,  with 
radius  r  >  0.  The  deterministic  part  of  (44)  has  been  termed  'norm-in¬ 
variant'  .  Let  the  components  of  z  be  independent.  The  Liapunov  function 
which  is  the  minimal  cost  of  transferring  x  =  xq  to  the  origin,  for  the 
deterministic  problem,  is 

Vx(x)  =  Ml  =  (x'x)1/2  . 


We  have 


Aw  “  RT  +  5RT 


f 


which  is  minimized  by 

(45)  u  =  -px/||xj|  , 
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the  optimal  deterministic  control,  and 
(46)  lV^x)  =  -p  +  (s-l)a2/2||xli, 

where  s  is  the  dimension  of  x.  If  the  target  set  has  a  radius  at 
2 

least  a  /2p,  then  Theorem  1  yields 


V^x)  -  r  <  C(x)  , 

The  fact  rhat  (46)  is  still  a  function  of  jjxj|  suggests  that  C(x) 
is  a  function  of  ||xj|.  Let  us  try 

(47)  V(x)  =  ||x)|  +  a  log  x'x  +  c. 


where  a  and  c  are  constants.  (47)  is  suggested  by  the  form  of  (46). 

(It  is  also  suggested  by  the  observation  that  the  'deterministic'  contribu¬ 
tion  to  l’V,  of  log  x'x,  2Ax’/x'x,  is  of  the  proper  form  to  cancel  part  of  the 
'stochastic'  contribution  of  V^(x)  to  L^V,  which  is  (s-l)  0^/211  xj| . ) 


LUV(  x)  = 


A  **  , 
PT 


+  -^7--  + 

x'  X 


2, 

Off  ( s- 


With  (45), 


LuV(x) 


Let  s  =  2  and  a  =  c2/2p,  then  LUV(x)  -  -p.  At  the  target  set  boundary. 
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2 

dS  =  {x:  x* x  =  r  },  we  have  V(x)  =  0.  Thus,  for  arbitrary  r  >  0,  set 
c  =  -r- a  log  (r2) . 

Now,  V(x)  >  0  and  L^x)  =  -p  in  X  -  S,  and 
CU(x)  =  V(x)  =  EU  J  UpdT  =  EUt  . 

Also,  since  (45)  minimizes  LUV(x)  over  all  admissible  controls,  by 
Corollary  1,  (45)  is  the  average- time- optimal  control  over  the  class  of 
admissible  controls.  If  s  >  2,  the  procedure  may  be  repeated.  This  will 
be  developed  elsewhere. 

Example  6.  Take  the  scalar  case 


dx  =  -xdt  +  udt  +  adz 

M  *  1,  S  =  (0), 

CU(x)  =  EjJ  /QU  (k  +  |  u|  )dt. 

The  optimal  deterministic  solution  is  (the  deterministic  version  is  a  problem 
in  [21]) 

f  V' (x)  *  (k  +  l)log(|x|  +  1),  |x|  <  k,  u  =  -  sign  x 

(48)  CAx) 

^  V"(x)  =  (k  +  l)log(k+l)log|  x| ,  I  x|  >  k,  u  =  0. 

log  k 

(49)  LV(x)  =  -(k+1)  -  a2(k+l)/2(|x|  +  l)2,  u  =  -  sign  x 
LV ( x)  =  -k  -  a2k/x2  ,  u  =  0. 
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of  the  quadratic  form 

29  5rd  line  below  (kj)  :  Replace  2Ax'/x'x  by  -2p/||  x||  . 
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At  j  xj  =  k,  C^(x)  does  not  have  a  derivative.  This  is  not  important 
in  the  scalar  case,  (it  can  be  assumed  that  a  (x)  satisfies  (2)  and  is 
zero  in  a  small  neighborhood  of  |  x|  =  k,  with  an  insignificant  change  in 
the  process.)  Since  LUC^(x)  <  -  k(x,  u)  for  j x|  ^  k,  Theorem  1  yields 

cU(x)  <  Cj(x)  . 

The  loss  for  the  stochastic  problem  is  less  than  C^(x),  since  the 
problem  is  scalar  and  C,(x)  is  convex  downward.  Such  an  improvement  is 

— Q 

uncommon  for  vector  problems. 

By  Corollary  1,  if  V(x)  and  u  satisfied  I/^x)  +  k  +  |  uj  =0  and 
LU  V(x)  +  k  +  |  u' |  £  0  for  u'  ^  u,  then  u  is  an  optimal  control  and 
V(x)  =  C(x).  Then,  u  must  satisfy 

(50)  us-  sign  dV/dx,  dV/dx  >  1 

u  =  0  ,  otherwise, 


exactly  the  form  of  the  deterministic  optimal  control.  The  form  (50)  is 
not  admissible,  but  may  be  approximated  arbitrarily  closely  by  an  admis¬ 
sible  control.  Since  the  problem  can  be  well,  defined  and  solvable  with  a 
slight  modification  of  cr  (x),  the  inadmissibility  will  he  ignored.  Since 
Cu(x)  <  V' (  x),  |  x|  <  k,  it  is  suggested  that  jdC(x)/dx|  <  jdV' (x)/dx|, 
i  xj  <  k,  and,  hence,  that  the  optimal  control  would  be  of  the  form 

sign  x,  j  xj  <  k*  <  k 
■i  =  0,  j  xj  i  k' . 


TGI*** 
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The  qualitative  information  inferred  above  can  be  substantiated  by- 
solving  tbe  exact  stochastic  problem  (which  is  easy  and  will  not  be  done  here). 

Define  S^.  =  (x:  |  x|  S  y},  X  >  0  and  let  X  be  a  large  set  con¬ 
taining  the  origin  with  cT  =  0  outside  X.  Now,  since  L°  log  (l  +  |  x|  )  <0 
in  X  -  S  for  any  X  >  - ,  (10)  yields 

P[  sup  log(  1  +  \  y..  \ )  *  X]  £  log  (1  +  |  x|  )/X. 

t  St50 
o 

Better  bounds  can  be  obtained  if  S^.  is  more  restricted.  Let  V{x)  =:  J  x|  n, 
n  Z  2.  Then 

L^x)  =  nj  x|  n"2  A(x) 

2  2 
\(x)  =  -x  +  ux  +  (n-l)cr  /2. 

If  A(x)  <0  in  X  -  S  ,  then 

P  [  sup  |  x  |  *  X]  £  |x|  n/Xn  . 

x  £t£G 
u 

The  smallest  S^.  (such  that  l'V  <0  in  X  -  and  [  uj  *  1)  corresponds 
to 

X  =  r-j  =  [-1  +  (1  +  2(n-l)a2)1/2]/2  , 

2  2 

and  then  we  require  u  =  -  sign  x.  for  X-j.  §  j  xj  £  y  f  where  XQ  =  (n-l)a  / 2. 
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